The present invention relates to a user interface for a text processing system or a graphics editing system utilizing speech input/output built in an OA equipment such as a personal computer, a workstation or a word processor, or a speech application system having a screen for displaying an operation result.
In a related system as disclosed in U.S. Pat. No. 5,600,765, entitled xe2x80x9cDisplay System Capable of Accepting User Commands by Use of Voice and Gesture Inputsxe2x80x9d, a user is required to reenter whole information even if only a portion of the input information is to be corrected, and a help function to show an alternative when the user points an error position is not provided. Namely, in the prior art, if an input by the user is an executable command to the system, the system executes the command as it is. If a command which is not executable by the system, for example, a copy command for a system having no copy function, is issued, a warning indicating non-executable is presented by the system. When user notices the warning, the user spontaneously executes a help system of the system. For an unpracticed user, it is difficult to refer explanation of a necessary help function. For a user input, the system always presents the same response without regard to the past system utilization status by the user. When a user interruption occurs during the operation of the system, the interruption is given a priority and the current operation is temporarily stopped to conduct the interruption.
In the prior art, the user must detect an error and when the user detects the error, the user must input whole information from the beginning. Further, since the response from the system is always constant without reared to the user status or the use environment of the system, the extent of the system operation state cannot be grasped and the user is hard to utilize the system.
It is an object of the present invention to provide an interface for text preparation, graphics edition and image edition which supports a user operation and which is user friendly, by allowing efficient input by speech input/output or finger pointing and displaying a 3-D (three-dimensional) CG (computer graphics) in a part of a screen under process to give a direction from the system to the user operation.
The above object of the present invention is achieved by a system which holds a user input content, predicts the next operation from the content and presents the operation.
The present invention provides a system which improves user operability and which is user friendly by substituting the system as if it were human beings through a human image displayed on a screen to convey a proposal by the system to create an environment of dialogue with the human.
The present invention comprises the following elements:
Input means includes a speech input device, an image input device (camera), a touch panel (pointing device) associated with coordinate positions of a display screen and an eye tracking input device for reading an eye line position of a user on the screen as a coordinate. Information is read in from those various types of input means to execute the system operation.
Output means includes a speech output device (speaker) and an image output device having a display screen. The display screen has a partial image display area for mainly displaying a system status in addition to an object to be processed. A human image is displayed in the image display area and a manner of display of the human image is changed in accordance with a past user input state.
The system holds a history of the user input command content and input times. When the same command content continues more than a predetermined number of times, the system automatically notifies the continuation of that command content through the human image displayed on the partial image display area. Specifically, an action which simulates the dialogue with the user and a speech output synchronized with the action are presented.
If a next input does not appear after the elapse of a predetermined time period from the immediately previous input time, the following input is prompted by using the human image displayed in the partial image display area.
When a speech input from the user cannot be correctly recognized and a correction command is issued to the system, the system expresses apology by using the human image displayed in the partial image display area.
Still further advantages of the present invention will become apparent to those of ordinarily skilled in the art upon reading and understanding the following detailed description of the preferred and alternate embodiments.